Initial boilerplate & working impl - Mayur Bhavsar - #20

The solution keeps a data lake and data warehouse safely synchronized with a transactional source while preserving complete history and supporting near real-time reporting.

1. Source

• SQLite database with 6 order-to-cash tables.
• Uses real primary keys, foreign keys, CHECK constraints, unique constraints, and indexes.
• Payment records do not cascade on order deletion, ensuring financial data is retained.

2. Change Data Capture

• 18 AFTER INSERT, UPDATE, and DELETE triggers write changes to cdc_events.
• Each event stores full before and after images.
• event_id is an auto-incrementing replay offset.
• Database-level capture prevents applications from bypassing CDC and ensures deletes are captured.
• Recursive triggers are enabled to capture cascaded child deletes.
• CDC is abstracted behind a CdcSource interface, allowing future replacement with WAL or Debezium.

3. Ingestion

• Micro-batch flow: validate → read → write files → append manifest.
• Uses at-least-once delivery with idempotent (table_name, event_id) processing for effectively exactly-once results.
• Checkpoints are derived from the lake manifest to avoid drift.
• Crashes may cause duplicates, but never data loss; retries reuse the same batch ID.

4. Data Lake

• Immutable NDJSON files partitioned by table_name and ingest_date.
• A fsynced manifest controls which files are visible to readers.
• Partial files remain invisible until committed.
• Primary keys and before/after images are stored as JSON text, allowing source schema changes without changing lake storage.

5. Data Warehouse

• Built with DuckDB and dbt using 14 models.
• Flow: typed CDC events → deduplicated events → SCD2 history → current views → financial marts.
• Events are processed by event_id, not arrival time, ensuring deterministic replay.
• Deletes are stored as tombstones with the last known record values.

6. Time Travel and Restore

• Supports as_of(entity, timestamp) and as_of_version(entity, version).
• Version-based ranking handles multiple changes occurring within the same timestamp.
• Restored data is written back through the source database and captured through CDC again.

7. Schema Safety

• Table contracts define expected schemas.
• Compatible changes warn and continue; breaking changes stop ingestion.
• Breaking examples include type, key, nullability, and destructive changes.
• Renames are treated as drop-plus-add and require human review.
• Halt status is persisted and requires explicit operator acknowledgement before restart.

8. Validation and Catalog

• Automated checks validate not-null, enums, uniqueness, and relationships.
• Business rules are re-tested in dbt.
• A catalog documents 15 datasets, including ownership, lineage, PII, consumers, cadence, and access paths.
• Catalog and warehouse are validated against each other to prevent undocumented datasets.